System and method for visualizing connected temporal and spatial information as an integrated visual representation on a user interface

ABSTRACT

As data about events and objects become more commonly available, analyzing and understanding of interrelated temporal and spatial information is increasingly a concern for military commanders, intelligence analysts and business analysts. A system and method is provided for creating a multidimensional visual representation of a group of data elements having integrated temporal and spatial properties. The data elements are included in the visual representation as corresponding visual elements, such that the data elements of the group linked by at least one association. The system includes a visualization manager for assembling the group of data elements using the at least one association and for assigning a connection visual element in the visual representation between a first visual element representing a first data element of the group and a second visual element representing a second data element of the group. The system also has a spatial visualization component, such as a sprite, configured for generating a spatial domain of the visual representation to include a reference surface for providing a spatial reference frame having at least two spatial dimensions. The reference surface is for relating the first visual element to a first location of interest in the spatial reference frame and for relating the second visual element to a second location of interest in the spatial reference frame. The system also has a temporal visualization component, such as a sprite, configured for generating a temporal domain of the visual representation operatively coupled to the spatial domain, the temporal domain for providing a common temporal reference frame for the locations of interest. The temporal domain includes a first time track, such as a timeline, coupled to the first location of interest and a second time track coupled to the second location of interest, such that the first visual element is positioned on the first time track and the second visual element is positioned on the second time track. Each of the time tracks configured for visually representing a respective temporal sequence of a plurality of the data elements at each of the locations of interest of the reference surface. In implementation of the method, the connection visual element represents a distributed association in at least one of the domains between the first visual element and the second visual element such that the visual representation is displayed on a user interface for subsequent interaction with user events, including animation of the visual elements to help in the analysis of the data contained in the visual representation.

This application claims the benefit of earlier filed Canadian Patentapplication No. (Not Yet Known) filed Mar. 15, 2004.

BACKGROUND OF THE INVENTION

The present invention relates to an interactive visual presentation ofmultidimensional data on a user interface.

Tracking and analyzing entities and streams of events, has traditionallybeen the domain of investigators, whether that be national intelligenceanalysts, police services or military intelligence. Business users alsoanalyze events in time and location to better understand phenomenon suchas customer behavior or transportation patterns. As data about eventsand objects become more commonly available, analyzing and understandingof interrelated temporal and spatial information is increasingly aconcern for military commanders, intelligence analysts and businessanalysts. Localized cultures, characters, organizations and theirbehaviors play an important part in planning and mission execution. Insituations of asymmetric warfare and peacekeeping, tracking relativelysmall and seemingly unconnected events over time becomes a means fortracking enemy behavior. For business applications, tracking ofproduction process characteristics can be a means for improving plantoperations. A generalized method to capture and visualize thisinformation over time for use by business and military applications,among others, is needed.

Many visualization techniques and products for analyzing complex eventinteractions only display information along a single dimension,typically one of time, geography or a network connectivity diagram. Eachof these types of visualizations is common and well understood. Forexample a Time-focused scheduling chart such as Microsoft (MS) Projectdisplays various project events over the single dimension of time, and aGeographic Information System (GIS) product, such as MS MapPoint, orESRI ArcView, is good for showing events in the single dimension oflocations on a map. There are also link analysis tools, such as Netmap(www.netmapanalytics.com) or Visual Analytics (www.visualanalytics.com)that display events as a network diagram, or graph, of objects andconnections between objects. Some of these systems are capable of usinganimation to display another dimension, typically time. Time is playedback, or scrolled, and the related spatial image display changes toreflect the state of information at a moment in time. However thistechnique relies on limited human short term memory to track and thenretain temporal changes and patterns in the spatial domain. Anothervisualization technique called “small multiples” uses repeated frames ofa condition or chart, each capturing an increment moment in time, muchlike looking at sequence of frames from a film laid side by side. Eachimage must be interpreted separately, and side-by-side comparisons made,to detect differences. This technique is expensive in terms of visualspace since an image must be generated for each moment of interest,which can be problematic when trying to simultaneously display multipleimages of adequate size that contain complex data content.

A technique has been developed, as described in InteractiveVisualization of Spatiotemporal Patterns using Spirals on a GeographicalMap—by Hewagamage et al. that uses spiral shaped ribbons as timelines toshow isolated sequences of events that have occurred at discretelocations on a geographical map. This technique is limited because ituses spiral timelines exclusively to show the periodic quality ofcertain types of events, while does not show connectivity between thetemporal and spatial information of data objects at multi-locationswithin the spatial domain. Further, event data objects placed on thespirals can suffer from occlusion, thereby providing for only a limitednumber of events and locations viewable with the spiral timelines.

It is an object of the present invention to provide a system and methodfor the integrated, interactive visual representation of a plurality ofevents and objects with spatial and temporal properties to obviate ormitigate at least some of the above-mentioned disadvantages.

SUMMARY OF THE INVENTION

Tracking and analyzing entities and streams of events, has traditionallybeen the domain of investigators, whether that be national intelligenceanalysts, police services or military intelligence. Business users alsoanalyze events in time and location to better understand phenomenon suchas customer behavior or transportation patterns. As data about eventsand objects become more commonly available, analyzing and understandingof interrelated temporal and spatial information is increasingly aconcern for military commanders, intelligence analysts and businessanalysts. Contrary to present analysis tools, a system and method isprovided for creating a multidimensional visual representation of agroup of data elements having integrated temporal and spatialproperties. The data elements are included in the visual representationas corresponding visual elements, such that the data elements of thegroup linked by at least one association. The system includes avisualization manager for assembling the group of data elements usingthe at least one association and for assigning a connection visualelement in the visual representation between a first visual elementrepresenting a first data element of the group and a second visualelement representing a second data element of the group. The system alsohas a spatial visualization component, such as a sprite, configured forgenerating a spatial domain of the visual representation to include areference surface for providing a spatial reference frame having atleast two spatial dimensions. The reference surface is for relating thefirst visual element to a first location of interest in the spatialreference frame and for relating the second visual element to a secondlocation of interest in the spatial reference frame. The system also hasa temporal visualization component, such as a sprite, configured forgenerating a temporal domain of the visual representation operativelycoupled to the spatial domain, the temporal domain for providing acommon temporal reference frame for the locations of interest. Thetemporal domain includes a first time track, such as a timeline, coupledto the first location of interest and a second time track coupled to thesecond location of interest, such that the first visual element ispositioned on the first time track and the second visual element ispositioned on the second time track. Each of the time tracks configuredfor visually representing a respective temporal sequence of a pluralityof the data elements at each of the locations of interest of thereference surface. In implementation of the method, the connectionvisual element represents a distributed association in at least one ofthe domains between the first visual element and the second visualelement such that the visual representation is displayed on a userinterface for subsequent interaction with user events, includinganimation of the visual elements to help in the analysis of the datacontained in the visual representation.

According to the present invention there is provided a method forcreating a multidimensional visual representation of a group of dataelements having integrated temporal and spatial properties, the dataelements being included in the visual representation as correspondingvisual elements, the data elements of the group linked by at least oneassociation, the method comprising the steps of: assembling the group ofdata elements using the at least one association; generating a spatialdomain of the visual representation to include a reference surface forproviding a spatial reference frame having at least two spatialdimensions, the reference surface for relating a first visual elementrepresenting a first data element of the group to a first location ofinterest in the spatial reference frame and relating a second visualelement representing a second data element of the group to a secondlocation of interest in the spatial reference frame; generating atemporal domain of the visual representation operatively coupled to thespatial domain, the temporal domain for providing a common temporalreference frame for the locations of interest, the temporal domainincluding a first time track coupled to the first location of interestand a second time track coupled to the second location of interest, thefirst visual element positioned on the first time track and the secondvisual element positioned on the second time track, each of the timetracks configured for visually representing a respective temporalsequence of a plurality of the data elements at each of the locations ofinterest of the reference surface; and assigning a connection visualelement in the visual representation between the first visual elementand the second visual element, the connection visual element forrepresenting a distributed association in at least one of the domainsbetween the first visual element and the second visual element; whereinthe visual representation is displayed on a user interface forsubsequent interaction with user events.

According to a further aspect of the present invention there is provideda system for creating a multidimensional visual representation of agroup of data elements having integrated temporal and spatialproperties, the data elements being included in the visualrepresentation as corresponding visual elements, the data elements ofthe group linked by at least one association, the system comprising: avisualization manager for assembling the group of data elements usingthe at least one association and for assigning a connection visualelement in the visual representation between a first visual elementrepresenting a first data element of the group and a second visualelement representing a second data element of the group; a spatialvisualization component configured for generating a spatial domain ofthe visual representation to include a reference surface for providing aspatial reference frame having at least two spatial dimensions, thereference surface for relating the first visual element to a firstlocation of interest in the spatial reference frame and relating thesecond visual element to a second location of interest in the spatialreference frame; and a temporal visualization component configured forgenerating a temporal domain of the visual representation operativelycoupled to the spatial domain, the temporal domain for providing acommon temporal reference frame for the locations of interest, thetemporal domain including a first time track coupled to the firstlocation of interest and a second time track coupled to the secondlocation of interest, the first visual element positioned on the firsttime track and the second visual element positioned on the second timetrack, each of the time tracks configured for visually representing arespective temporal sequence of a plurality of the data elements at eachof the locations of interest of the reference surface; and wherein theconnection visual element represents a distributed association in atleast one of the domains between the first visual element and the secondvisual element such that the visual representation is displayed on auser interface for subsequent interaction with user events.

According to a still further aspect of the present invention there isprovided a computer program product for creating a multidimensionalvisual representation of a group of data elements having integratedtemporal and spatial properties, the data elements being included in thevisual representation as corresponding visual elements, the dataelements of the group linked by at least one association, the computerprogram product comprising: a computer readable medium; a visualizationmodule stored on the computer readable medium for assembling the groupof data elements using the at least one association and for assigning aconnection visual element in the visual representation between a firstvisual element representing a first data element of the group and asecond visual element representing a second data element of the group; aspatial visualization module stored on the computer readable medium forgenerating a spatial domain of the visual representation to include areference surface for providing a spatial reference frame having atleast two spatial dimensions, the reference surface for relating thefirst visual element to a first location of interest in the spatialreference frame and relating the second visual element to a secondlocation of interest in the spatial reference frame; and a temporalvisualization module stored on the computer readable medium forgenerating a temporal domain of the visual representation operativelycoupled to the spatial domain, the temporal domain for providing acommon temporal reference frame for the locations of interest, thetemporal domain including a first time track coupled to the firstlocation of interest and a second time track coupled to the secondlocation of interest, the first visual element positioned on the firsttime track and the second visual element positioned on the second timetrack, each of the time tracks configured for visually representing arespective temporal sequence of a plurality of the data elements at eachof the locations of interest of the reference surface; wherein theconnection visual element represents a distributed association in atleast one of the domains between the first visual element and the secondvisual element such that the visual representation is displayed on auser interface for subsequent interaction with user events.

BRIEF DESCRIPTION OF THE DRAWINGS

A better understanding of these and other embodiments of the presentinvention can be obtained with reference to the following drawings anddetailed description of the preferred embodiments, in which:

FIG. 1 is a block diagram of a data processing system for avisualization tool;

FIG. 2 shows further details of the data processing system of FIG. 1;

FIG. 3 shows further details of the visualization tool of FIG. 1;

FIG. 4 shows further details of a visualization representation fordisplay on a visualization interface of the system of FIG. 1;

FIG. 5 is an example visualization representation of FIG. 1 showingEvents in Concurrent Time and Space;

FIG. 6 shows example data objects and associations of FIG. 1;

FIG. 7 shows further example data objects and associations of FIG. 1;

FIG. 8 shows changes in orientation of a reference surface of thevisualization representation of FIG. 1;

FIG. 9 is an example timeline of FIG. 8;

FIG. 10 is a further example timeline of FIG. 8;

FIG. 11 is a further example timeline of FIG. 8 showing a time chart;

FIG. 12 is a further example of the time chart of FIG. 11;

FIG. 13 shows example user controls for the visualization representationof FIG. 5;

FIG. 14 shows an example operation of the tool of FIG. 3;

FIG. 15 shows a further example operation of the tool of FIG. 3;

FIG. 16 shows a further example operation of the tool of FIG. 3;

FIG. 17 shows an example visualization representation of FIG. 4containing events and target tracking over space and time showingconnections between events;

FIG. 18 shows an example visualization representation containing eventsand target tracking over space and time showing connections betweenevents on a time chart of FIG. 11, and

FIG. 19 is an example operation of the visualization tool of FIG. 3.

It is noted that similar references are used in different figures todenote similar components.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The following detailed description of the embodiments of the presentinvention does not limit the implementation of the invention to anyparticular computer programming language. The present invention may beimplemented in any computer programming language provided that the OS(Operating System) provides the facilities that may support therequirements of the present invention. A preferred embodiment isimplemented in the Java computer programming language (or other computerprogramming languages in conjunction with C/C++). Any limitationspresented would be a result of a particular type of operating system,computer programming language, or data processing system and would notbe a limitation of the present invention.

Visualization Environment

Referring to FIG. 1, a visualization data processing system 100 includesa visualization tool 12 for processing a collection of data objects 14as input data elements to a user interface 202. The data objects 14 arecombined with a respective set of associations 16 by the tool 12 togenerate an interactive visual representation 18 on the visual interface(VI) 202. The data objects 14 include event objects 20, location objects22, and entity objects 24, as further described below. The set ofassociations 16 include individual associations 26 that associatetogether various subsets of the objects 20, 22, 24, as further describedbelow. Management of the data objects 14 and set of associations 16 aredriven by user events 109 of a user (not shown) during interaction withthe visual representation 18.

Data Processing System

Referring to FIG. 2, the data processing system 100 has a user interface108 for interacting with the tool 12, the user interface 108 beingconnected to a memory 102 via a BUS 106. The interface 108 is coupled toa processor 104 via the BUS 106, to interact with user events 109 tomonitor or otherwise instruct the operation of the tool 12 via anoperating system 110. The user interface 108 can include one or moreuser input devices such as but not limited to a QWERTY keyboard, akeypad, a trackwheel, a stylus, a mouse, and a microphone. The visualinterface 202 is considered the user output device, such as but notlimited to a computer screen display. If the screen is touch sensitive,then the display can also be used as the user input device as controlledby the processor 104. Further, it is recognized that the data processingsystem 100 can include a computer readable storage medium 46 coupled tothe processor 104 for providing instructions to the processor 104 and/orthe tool 12. The computer readable medium 46 can include hardware and/orsoftware such as, by way of example only, magnetic disks, magnetic tape,optically readable medium such as CD/DVD ROMS, and memory cards. In eachcase, the computer readable medium 46 may take the form of a small disk,floppy diskette, cassette, hard disk drive, solid-state memory card, orRAM provided in the memory 102. It should be noted that the above listedexample computer readable mediums 46 can be used either alone or incombination.

Referring again to FIG. 2, the tool 12 interacts via link 116 with a VImanager 112 (also known as a visualization renderer) of the system 100for presenting the visual representation 18 on the visual interface 202.The tool 12 also interacts via link 118 with a data manager 114 of thesystem 100 to coordinate management of the data objects 14 andassociation set 16 from data files or tables 122 of the memory 102. Itis recognized that the objects 14 and association set 16 could be storedin the same or separate tables 122, as desired. The data manager 114 canreceive requests for storing, retrieving, amending, or creating theobjects 14 and association set 16 via the tool 12 and/or directly vialink 120 from the VI manager 112, as driven by the user events 109and/or independent operation of the tool 12. The data manager 114manages the objects 14 and association set 16 via link 123 with thetables 122. Accordingly, the tool 12 and managers 112, 114 coordinatethe processing of data objects 14, association set 16 and user events109 with respect to the content of the screen representation 18displayed in the visual interface 202.

Tool Information Model

Referring to FIG. 1, a tool information model is composed of the fourbasic data elements (objects 20, 22, 24 and associations 26) that canhave corresponding display elements in the visual representation 18. Thefour elements are used by the tool 12 to describe interconnectedactivities and information in time and space as the integrated visualrepresentation 18, as further described below.

Event Data Objects 20

Events are data objects 20 that represent any action that can bedescribed. The following are examples of events;

-   -   Bill was at Toms house at 3 pm,    -   Tom phoned Bill on Thursday,    -   A tree fell in the forest at 4:13 am, Jun. 3, 1993 and    -   Tom will move to Spain in the summer of 2004.        The Event is related to a location and a time at which the        action took place, as well as several data properties and        display properties including such as but not limited to; a short        text label, description, location, start-time, end-time, general        event type, icon reference, visual layer settings, priority,        status, user comment, certainty value, source of information,        and default+user-set color. The event data object 20 can also        reference files such as images or word documents.

Locations and times may be described with varying precision. Forexample, event times can be described as “during the week of January5^(th)” or “in the month of September”. Locations can be described as“Spain” or as “New York” or as a specific latitude and longitude.

Entity Data Objects 24

Entities are data objects 24 that represent any thing related to orinvolved in an event, including such as but not limited to; people,objects, organizations, equipment, businesses, observers, affiliationsetc. Data included as part of the Entity data object 24 can be shorttext label, description, general entity type, icon reference, visuallayer settings, priority, status, user comment, certainty value, sourceof information, and default+user-set color. The entity data can alsoreference files such as images or word documents. It is recognized inreference to FIGS. 6 and 7 that the term Entities includes “People”, aswell as equipment (e.g. vehicles), an entire organization (e.g.corporate entity), currency, and any other object that can be trackedfor movement in the spatial domain 400. It is also recognized that theentities 24 could be stationary objects such as but not limited tobuildings. Further, entities can be phone numbers and web sites. To beexplicit, the entities 24 as given above by example only can be regardedas Actors

Locations Data Objects 22

Locations are data objects 22 that represent a place within a spatialcontext/domain, such as a geospatial map, a node in a diagram such as aflowchart, or even a conceptual place such as “Shang-ri-la” or other“locations” that cannot be placed at a specific physical location on amap or other spatial domain. Each Location data object 22 can store suchas but not limited to; position coordinates, a label, description, colorinformation, precision information, location type, non-geospatial flagand user comments.

Associations

Event 20, Location 22 and Entity 24 are combined into groups or subsetsof the data objects 14 in the memory 102 (see FIG. 2) using associations26 to describe real-world occurrences. The association is defined as aninformation object that describes a pairing between 2 data objects 14.For example, in order to show that a particular entity was present whenan event occurred, the corresponding association 26 is created torepresent that Entity X “was present at” Event A. For example,associations 26 can include such as but not limited to; describing acommunication connection between two entities 24, describing a physicalmovement connection between two locations of an entity 24, and arelationship connection between a pair of entities 24 (e.g. familyrelated and/or organizational related). It is recognised that theassociations 26 can describe direct and indirect connections. Otherexamples can include phone numbers and web sites.

Visualization Tool 12

Referring to FIG. 3, the visualization tool 12 has a visualizationmanager 300 for interacting with the data objects 14 for presentation tothe interface 202 via the VI manager 112. The Data Objects 14 are formedinto groups 27 through the associations 26 and processed by theVisualization Manager 300. The groups 27 comprise selected subsets ofthe objects 20, 22, 24 combined via selected associations 26. Thiscombination of data objects 14 and association sets 16 can beaccomplished through predefined groups 27 added to the tables 122 and/orthrough the user events 109 during interaction of the user directly withselected data objects 14 and association sets 16 via the controls 306.It is recognized that the predefined groups 27 could be loaded into thememory 102 (and tables 122) via the computer readable medium 46 (seeFIG. 2). The Visualization manager 300 also processes user event 109input through interaction with a time slider and other controls 306,including several interactive controls for supporting navigation andanalysis of information within the visual representation 18 (see FIG. 1)as further described below.

The Visualization Manager 300 processes the translation from raw dataobjects 14 to the visual representation 18. First, Data Objects 14 andassociations 16 are formed by the Visualization Manager 300 into thegroups 27, as noted in the tables 122, and then processed. TheVisualization Manager 300 matches the raw data objects 14 andassociations 16 with sprites 308 (i.e. visual processingobjects/components that know how to draw and render visual elements forspecified data objects 14 and associations 16) and sets a drawingsequence for implementation by the VI manager 112. The sprites 308 arevisualization components that take predetermined information schema asinput and output graphical elements such as lines, text, images andicons to the computers graphics system. Entity 24, event 20 and location22 data objects each can have a specialized sprite 308 type designed torepresent them. A new sprite instance is created for each entity, eventand location instance to manage their representation in the visualrepresentation 18 on the display.

The sprites 308 are processed in order by the visualization manager 300,starting with the spatial domain (terrain) context and locations,followed by Events and Timelines, and finally Entities. Timelines aregenerated and Events positioned along them. Entities are rendered lastby the sprites 308 since the entities depend on Event positions. It isrecognised that processing order of the sprites 308 can be other than asdescribed above.

The Visualization manager 112 renders the sprites 308 to create thefinal image including visual elements representing the data objects 14and associates 16 of the groups 27, for display as the visualrepresentation 18 on the interface 202. After the visual representation18 is on the interface 202, the user event 109 inputs flow into theVisualization Manager, through the VI manager 112 and cause the visualrepresentation 18 to be updated. The Visualization Manager 300 can beoptimized to update only those sprites 308 that have changed in order tomaximize interactive performance between the user and the interface 202.

Layout of the Visualization Representation 18

The visualization technique of the visualization tool 12 is designed toimprove perception of entity activities, movements and relationships asthey change over time in a concurrent time-geographic ortime-diagrammatical context. The visual representation 18 of the dataobjects 14 and associations 16 consists of a combined temporal-spatialdisplay to show interconnecting streams of events over a range of timeon a map or other schematic diagram space, both hereafter referred to incommon as a spatial domain 400 (see FIG. 4). Events can be representedwithin an X,Y,T coordinate space, in which the X,Y plane shows thespatial domain 400 (e.g. geographic space) and the Z-axis represents atime series into the future and past, referred to as a temporal domain402. In addition to providing the spatial context, a reference surface(or reference spatial domain) 404 marks an instant of focus betweenbefore and after, such that events “occur” when they meet the surface ofthe ground reference surface 404. FIG. 4 shows how the visualizationmanager 300 (see FIG. 3) combines individual frames 406 (spatial domains400 taken at different times Ti 407) of event/entity/location visualelements 410, which are translated into a continuous integrated spatialand temporal visual representation 18. It should be noted connectionvisual elements 412 can represent presumed location (interpolated) ofEntity between the discrete event/entity/location represented by thevisual elements 410. Another interpretation for connections elements 412could be signifying communications between different Entities atdifferent locations, which are related to the same event as furtherdescribed below.

Referring to FIG. 5, an example visual representation 18 visuallydepicts events over time and space in an x, y, t space (or x, y, z, tspace with elevation data). The example visual representation 18generated by the tool 12 (see FIG. 2) is shown having the time domain402 as days in April, and the spatial domain 400 as a geographical mapproviding the instant of focus (of the reference surface 404) assometime around noon on April 23—the intersection point between thetimelines 422 and the reference surface 404 represents the instant offocus. The visualization representation 18 represents the temporal 402,spatial 400 and connectivity elements 412 (between two visual elements410) of information within a single integrated picture on the interface202 (see FIG. 1). Further, the tool 12 provides an interactive analysistool for the user with interface controls 306 to navigate the temporal,spatial and connectivity dimensions. The tool 12 is suited to theinterpretation of any information in which time, location andconnectivity are key dimensions that are interpreted together. Thevisual representation 18 is used as a visualization technique fordisplaying and tracking events, people, and equipment within thecombined temporal and spatial domains 402, 400 display. The visualrepresentation 18 can be applied as an analyst workspace forexploration, deep analysis and presentation for such as but not limitedto:

-   -   Situations involving people and organizations that interact over        time and in which geography or territory plays a role;    -   Storing and reviewing activity reports over a given period. Used        in this way the representation 18 could provide a means to        determine a living history, context and lessons learned from        past events; and    -   As an analysis and presentation tool for long term tracking and        surveillance of persons and equipment activities.

The visualization tool 12 provides the visualization representation 18as an interactive display, such that the users (e.g. intelligenceanalysts, business marketing analysts) can view, and work with, largenumbers of events. Further, perceived patterns, anomalies andconnections can be explored and subsets of events can be grouped into“story” or hypothesis fragments. The visualization tool 12 includes avariety of capabilities such as but not limited to:

-   -   An event-based information architecture with places, events,        entities (e.g. people) and relationships;    -   Past and future time visibility and animation controls;    -   Data input wizards for describing single events and for loading        many events from a table;    -   Entity and event connectivity analysis in time and geography;    -   Path displays in time and geography;    -   Configurable workspaces allowing ad hoc, drag and drop        arrangements of events;    -   Search, filter and drill down tools;    -   Creation of sub-groups and overlays by selecting events and        dragging them into sets (along with associated spatial/time        scope properties); and    -   Adaptable display functions including dynamic show/hide        controls.        Example Objects 14 With Associations 16

In the visualization tool 12, specific combinations of associated dataelements (objects 20, 22, 24 and associations 26) can be defined. Thesedefined groups 27 are represented visually as visual elements 410 inspecific ways to express various types of occurrences in the visualrepresentation 18. The following are examples of how the groups 27 ofassociated data elements can be formed to express specific occurrencesand relationships shown as the connection visual elements 412.

Referring to FIGS. 6 and 7, example groups 27 (denoting common realworld occurrences) are shown with selected subsets of the objects 20,22, 24 combined via selected associations 26. The correspondingvisualization representation 18 is shown as well including the temporaldomain 402, the spatial domain 400, connection visual elements 412 andthe visual elements 410 representing the event/entity/locationcombinations. It is noted that example applications of the groups 27 aresuch as but not limited to those shown in FIGS. 6 and 7. In the FIGS. 6and 7 it is noted that event objects 20 are labeled as “Event 1”, “Event2”, location objects 22 are labeled as “Location A”, “Location B”, andentity objects 24 are labeled as “Entity X”, “Entity Y”. The set ofassociations 16 are labeled as individual associations 26 withconnections labeled as either solid or dotted lines 412 between twoevents, or dotted in the case of an indirect connection between twolocations.

Visual Elements Corresponding to Spatial and Temporal Domains

The visual elements 410 and 412, their variations and behaviorfacilitate interpretation of the concurrent display of events in thetime 402 and space 400 domains. In general, events reference thelocation at which they occur and a list of Entities and their role inthe event. The time at which the event occurred or the time span overwhich the event occurred are stored as parameters of the event.

Spatial Domain Representation

Referring to FIG. 8, the primary organizing element of the visualizationrepresentation 18 is the 2D/3D spatial reference frame (subsequentlyincluded herein with reference to the spatial domain 400). The spatialdomain 400 consists of a true 2D/3D graphics reference surface 404 inwhich a 2D or 3 dimensional representation of an area is shown. Thisspatial domain 400 can be manipulated using a pointer device (notshown—part of the controls 306—see FIG. 3) by the user of the interface108 (see FIG. 2) to rotate the reference surface 404 with respect to aviewpoint 420 or viewing ray extending from a viewer 423. The user (i.e.viewer 423) can also navigate the reference surface 404 by scrolling inany direction, zooming in or out of an area and selecting specific areasof focus. In this way the user can specify the spatial dimensions of anarea of interest the reference surface 404 in which to view events intime. The spatial domain 400 represents space essentially as a plane(e.g. reference surface 404), however is capable of representing 3dimensional relief within that plane in order to express geographicalfeatures involving elevation. The spatial domain 400 can be madetransparent so that timelines 422 of the temporal domain 402 can extendbehind the reference surface 404 are still visible to the user. FIG. 8shows how the viewer 423 facing timelines 422 can rotate to face theviewpoint 420 no matter how the reference surface 404 is rotated in 3dimensions with respect to the viewpoint 420.

The spatial domain 400 includes visual elements 410, 412 (see FIG. 4)that can represent such as but not limited to map information, digitalelevation data, diagrams, and images used as the spatial context. Thesetypes of spaces can also be combined into a workspace. The user can alsocreate diagrams using drawing tools (of the controls 306—see FIG. 3)provided by the visualization tool 12 to create custom diagrams andannotations within the spatial domain 400.

Event Representation and Interactions

Referring to FIGS. 4 and 8, events are represented by a glyph, or iconas the visual element 410, placed along the timeline 422 at the point intime that the event occurred. The glyph can be actually a group ofgraphical objects, or layers, each of which expresses the content of theevent data object 20 (see FIG. 1) in a different way. Each layer can betoggled and adjusted by the user on a per event basis, in groups oracross all event instances. The graphical objects or layers for eventvisual elements 410 are such as but not limited to:

-   -   1. Text label        -   The Text label is a text graphic meant to contain a short            description of the event content. This text always faces the            viewer 423 no matter how the reference surface 404 is            oriented. The text label incorporates a de-cluttering            function that separates it from other labels if they            overlap. When two events are connected with a line (see            connections 412 below) the label will be positioned at the            midpoint of the connection line between the events. The            label will be positioned at the end of a connection line            that is clipped at the edge of the display area.    -   2. Indicator—Cylinder, Cube or Sphere        -   The indicator marks the position in time. The color of the            indicator can be manually set by the user in an event            properties dialog. Color of event can also be set to match            the Entity that is associated with it. The shape of the            event can be changed to represent different aspect of            information and can be set by the user. Typically it is used            to represent a dimension such as type of event or level of            importance.    -   3. Icon        -   An icon or image can also be displayed at the event            location. This icon may used to describe some aspect of the            content of the event. This icon may be user-specified or            entered as part of a data file of the tables 122 (see FIG.            2).    -   4. Connection elements 412        -   Connection elements 412 can be lines, or other geometrical            curves, which are solid or dashed lines that show            connections from an event to another event, place or target.            A connection element 412 may have a pointer or arrowhead at            one end to indicate a direction of movement, polarity,            sequence or other vector-like property. If the connected            object is outside of the display area, the connection            element 412 can be coupled at the edge of the reference            surface 404 and the event label will be positioned at the            clipped end of the connection element 412.    -   5. Time Range Indicator        -   A Time Range Indicator (not shown) appears if an event            occurs over a range of time. The time range can be shown as            a line parallel to the timeline 422 with ticks at the end            points. The event Indicator (see above) preferably always            appears at the start time of the event.

The Event visual element 410 can also be sensitive to interaction. Thefollowing user events 109 via the user interface 108 (see FIG. 2) arepossible, such as but not limited to:

Mouse-Left-Click:

Selects the visual element 410 of the visualization representation 18 onthe VI 202 (see FIG. 2) and highlights it, as well as simultaneouslydeselecting any previously selected visual element 410, as desired.

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click

Adds the visual element 410 to an existing selection set.

Mouse-Left-Double-Click:

Opens a file specified in an event data parameter if it exists. The filewill be opened in a system-specified default application window on theinterface 202 based on its file type.

Mouse-Right-Click:

Displays an in-context popup menu with options to hide, delete and setproperties.

Mouse over Drilldown:

When the mouse pointer (not shown) is placed over the indicator, a textwindow is displayed next to the pointer, showing information about thevisual element 410. When the mouse pointer is moved away from theindicator, the text window disappears.

Location Representation

Locations are visual elements 410 represented by a glyph, or icon,placed on the reference surface 404 at the position specified by thecoordinates in the corresponding location data object 22 (see FIG. 1).The glyph can be a group of graphical objects, or layers, each of whichexpresses the content of the location data object 22 in a different way.Each layer can be toggled and adjusted by the user on a per Locationbasis, in groups or across all instances. The visual elements 410 (e.g.graphical objects or layers) for Locations are such as but not limitedto:

-   -   1. Text Label        -   The Text label is a graphic object for displaying the name            of the location. This text always faces the viewer 422 no            matter how the reference surface 404 is oriented. The text            label incorporates a de-cluttering function that separates            it from other labels if they overlap.    -   2. Indicator        -   The indicator is an outlined shape that marks the position            or approximate position of the Location data object 22 on            the reference surface 404. There are, such as but not            limited to, 7 shapes that can be selected for the locations            visual elements 410 (marker) and the shape can be filled or            empty. The outline thickness can also be adjusted. The            default setting can be a circle and can indicate spatial            precision with size. For example, more precise locations,            such as addresses, are smaller and have thicker line width,            whereas a less precise location is larger in diameter, but            uses a thin line width.

The Location visual elements 410 are also sensitive to interaction. Thefollowing interactions are possible:

Mouse-Left-Click:

Selects the location visual element 410 and highlights it, whiledeselecting any previously selected location visual elements 410.

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click

Adds the location visual element 410 to an existing selection set.

Mouse-Left-Double-Click:

Opens a file specified in a Location data parameter if it exists. Thefile will be opened in a system-specified default application windowbased on its file type.

Mouse-Right-Click:

Displays an in-context popup menu with options to hide, delete and setproperties of the location visual element 410.

Mouseover Drilldown:

When the Mouse pointer is placed over the location indicator, a textwindow showing information about the location visual element 410 isdisplayed next to the pointer. When the mouse pointer is moved away fromthe indicator, the text window disappears.

Mouse-Left-Click-Hold-and-Drag:

Interactively repositions the location visual element 410 by dragging itacross the reference surface 404.

Non-Spatial Locations

Locations have the ability to represent indeterminate position. Theseare referred to as non-spatial locations. Locations tagged asnon-spatial can be displayed at the edge of the reference surface 404just outside of the spatial context of the spatial domain 400. Thesenon-spatial or virtual locations can be always visible no matter wherethe user is currently zoomed in on the reference surface 404. Events andTimelines 422 that are associated with non-spatial Locations can berendered the same way as Events with spatial Locations.

Entity Representation

Entity visual elements 410 are represented by a glyph, or icon, and canbe positioned on the reference surface 404 or other area of the spatialdomain 400, based on associated Event data that specifies its positionat the current Moment of Interest 900 (see FIG. 9) (i.e. specific pointon the timeline 422 that intersects the reference surface 404). If thecurrent Moment of Interest 900 lies between 2 events in time thatspecify different positions, the Entity position will be interpolatedbetween the 2 positions. Alternatively, the Entity could be positionedat the most recent known location on he reference surface 404. TheEntity glyph is actually a group of the entity visual elements 410 (e.g.graphical objects, or layers) each of which expresses the content of theevent data object 20 in a different way. Each layer can be toggled andadjusted by the user on a per event basis, in groups or across all eventinstances. The entity visual elements 410 are such as but not limitedto:

-   -   1. Text Label        -   The Text label is a graphic object for displaying the name            of the Entity. This text always faces the viewer no matter            how the reference surface 404 is oriented. The text label            incorporates a de-cluttering function that separates it from            other labels if they overlap.    -   2. Indicator        -   The indicator is a point showing the interpolated or real            position of the Entity in the spatial context of the            reference surface 404. The indicator assumes the color            specified as an Entity color in the Entity data model.    -   3. Image Icon        -   An icon or image is displayed at the Entity location. This            icon may used to represent the identity of the Entity. The            displayed image can be user-specified or entered as part of            a data file. The Image Icon can have an outline border that            assumes the color specified as the Entity color in the            Entity data model. The Image Icon incorporates a            de-cluttering function that separates it from other Entity            Image Icons if they overlap.    -   4. Past Trail        -   The Past Trail is the connection visual element 412, as a            series of connected lines that trace previous known            positions of the Entity over time, starting from the current            Moment of Interest 900 and working backwards into past time            of the timeline 422. Previous positions are defined as            Events where the Entity was known to be located. The Past            Trail can mark the path of the Entity over time and space            simultaneously.    -   5. Future Trail        -   The Future Trail is the connection visual element 412, as a            series of connected lines that trace future known positions            of the Entity over time, starting from the current Moment of            Interest 900 and working forwards into future time. Future            positions are defined as Events where the Entity is known to            be located. The Future Trail can mark the future path of the            Entity over time and space simultaneously.

The Entity representation is also sensitive to interaction. Thefollowing interactions are possible, such as but not limited to:

Mouse-Left-Click:

Selects the entity visual element 410 and highlights it and deselectsany previously selected entity visual element 410.

Ctrl-Mouse-Left-Click and Shift-Mouse-Left-Click

Adds the entity visual element 410 to an existing selection set

Mouse-Left-Double-C lick:

Opens the file specified in an Entity data parameter if it exists. Thefile will be opened in a system-specified default application windowbased on its file type.

Mouse-Right-Click:

Displays an in-context popup menu with options to hide, delete and setproperties of the entity visual element 410.

Mouseover Drilldown:

When the Mouse pointer is placed over the indicator, a text windowshowing information about the entity visual element 410 is displayednext to the pointer. When the mouse pointer is moved away from theindicator, the text window disappears.

Temporal Domain Including Timelines

Referring to FIGS. 8 and 9, the temporal domain provides a commontemporal reference frame for the spatial domain 400, whereby the domains400, 402 are operatively coupled to one another to simultaneouslyreflect changes in interconnected spatial and temporal properties of thedata elements 14 and associations 16. Timelines 422 (otherwise known astime tracks) represent a distribution of the temporal domain 402 overthe spatial domain 400, and are a primary organizing element ofinformation in the visualization representation 18 that make it possibleto display events across time within the single spatial display on theVI 202 (see FIG. 1). Timelines 422 represent a stream of time through aparticular Location visual element 410 a positioned on the referencesurface 404 and can be represented as a literal line in space. Otheroptions for representing the timelines/time tracks 422 are such as butnot limited to curved geometrical shapes (e.g. spirals) including 2D and3D curves when combining two or more parameters in conduction with thetemporal dimension. Each unique Location of interest (represented by thelocation visual element 410 a) has one Timeline 422 that passes throughit. Events (represented by event visual elements 410 b) that occur atthat Location are arranged along this timeline 422 according to theexact time or range of time at which the event occurred. In this waymultiple events (represented by respective event visual elements 410 b)can be arranged along the timeline 422 and the sequence made visuallyapparent. A single spatial view will have as many timelines 422 asnecessary to show every Event at every location within the currentspatial and temporal scope, as defined in the spatial 400 and temporal402 domains (see FIG. 4) selected by the user. In order to makecomparisons between events and sequences of event between locations, thetime range represented by multiple timelines 422 projecting through thereference surface 404 at different spatial locations is synchronized. Inother words the time scale is the same across all timelines 422 in thetime domain 402 of the visual representation 18. Therefore, it isrecognised that the timelines 422 are used in the visual representation18 to visually depict a graphical visualization of the data objects 14over time with respect to their spatial properties/attributes.

Representing Current, Past and Future

Three distinct strata of time are displayed by the timelines 422,namely;

-   -   1. The “moment of interest” 900 or browse time, as selected by        the user,    -   2. a range 902 of past time preceding the browse time called        “past”, and    -   3. a range 904 of time after the moment of interest 900, called        “future”

On a 3D Timeline 422, the moment of focus 900 is the point at which thetimeline intersects the reference surface 404. An event that occurs atthe moment of focus 900 will appear to be placed on the referencesurface 404 (event representation is described above). Past and futuretime ranges 902, 904 extend on either side (above or below) of themoment of interest 900 along the timeline 422. Amount of time into thepast or future is proportional to the distance from the moment of focus900. The scale of time may be linear or logarithmic in either direction.The user may select to have the direction of future to be down and pastto be up or vice versa.

There are three basic variations of Spatial Timelines 422 that emphasizespatial and temporal qualities to varying extents. Each variation has aspecific orientation and implementation in terms of its visualconstruction and behavior in the visualization representation 18 (seeFIG. 1). The user may choose to enable any of the variations at any timeduring application runtime, as further described below.

3D Z-axis Timelines

FIG. 10 shows how 3D Timelines 422 pass through reference surface 404locations 410 a. 3 D timelines 422 are locked in orientation (angle)with respect to the orientation of the reference surface 404 and areaffected by changes in perspective of the reference surface 404 aboutthe viewpoint 420 (see FIG. 8). For example, the 3D Timelines 422 can beoriented normal to the reference surface 404 and exist within itscoordinate space. Within the 3D spatial domain 400, the referencesurface 404 is rendered in the X-Y plane and the timelines 422 runparallel to the Z-axis through locations 410 a on the reference surface404. Accordingly, the 3D Timelines 422 move with the reference surface404 as it changes in response to user navigation commands and viewpointchanges about the viewpoint 420, much like flag posts are attached tothe ground in real life. The 3D timelines 422 are subject to the sameperspective effects as other objects in the 3D graphical window of theVI 202 (see FIG. 1) displaying the visual representation 18. The 3DTimelines 422 can be rendered as thin cylindrical volumes and arerendered only between events 410 a with which it shares a location andthe location 410 a on the reference surface 404. The timeline 422 mayextend above the reference surface 404, below the reference surface 404,or both. If no events 410 b for its location 410 a are in view thetimeline 422 is not shown on the visualization representation 18.

3D Viewer Facing Timelines

Referring to FIG. 8, 3D Viewer-facing Timelines 422 are similar to 3DTimelines 422 except that they rotate about a moment of focus 425 (pointat which the viewing ray of the viewpoint 420 intersects the referencesurface 404) so that the 3D Viewer-facing Timeline 422 always remainperpendicular to viewer 423 from which the scene is rendered. 3DViewer-facing Timelines 422 are similar to 3D Timelines 422 except thatthey rotate about the moment of focus 425 so that they are alwaysparallel to a plane 424 normal to the viewing ray between the viewer 423and the moment of focus 425. The effect achieved is that the timelines422 are always rendered to face the viewer 423, so that the length ofthe timeline 422 is always maximized and consistent. This techniqueallows the temporal dimension of the temporal domain 402 to be read bythe viewer 423 indifferent to how the reference surface 404 many beoriented to the viewer 423. This technique is also generally referred toas “billboarding” because the information is always oriented towards theviewer 423. Using this technique the reference surface 404 can be viewedfrom any direction (including directly above) and the temporalinformation of the timeline 422 remains readable.

Linked TimeChart Timelines

Referring to FIG. 11, showing how an overlay time chart 430 is connectedto the reference surface 404 locations 410 a by timelines 422. Thetimelines 422 of the Linked TimeChart 430 are timelines 422 that connectthe 2D chart 430 (e.g. grid) in the temporal domain 402 to locations 410a marked in the 3D spatial domain 400. The timeline grid 430 is renderedin the visual representation 18 as an overlay in front of the 2D or 3Dreference surface 404. The timeline chart 430 can be a rectangularregion containing a regular or logarithmic time scale upon which eventrepresentations 410 b are laid out. The chart 430 is arranged so thatone dimension 432 is time and the other is location 434 based on theposition of the locations 410 a on the reference surface 404. As thereference surface 404 is navigated or manipulated the timelines 422 inthe chart 430 move to follow the new relative location 410 a positions.This linked location and temporal scrolling has the advantage that it iseasy to make temporal comparisons between events since time isrepresented in a flat chart 430 space. The position 410 b of the eventcan always be traced by following the timeline 422 down to the referencesurface 404 to the location 410 a.

Referring to FIGS. 11 and 12, the TimeChart 430 can be rendered in 2orientations, one vertical and one horizontal. In the vertical mode ofFIG. 11, the TimeChart 430 has the location dimension 434 shownhorizontally, the time dimension 432 vertically, and the timelines 422connect vertically to the reference surface 404. In the horizontal modeof FIG. 12, the TimeChart 430 has the location dimension 434 shownvertically, the time dimension 432 shown horizontally and the timelines422 connect to the reference surface 404 horizontally. In both cases theTimeChart 430 position in the visualization representation 18 can bemoved anywhere on the screen of the VI 202 (see FIG. 1), so that thechart 430 may be on either side of the reference surface 404 or in frontof the reference surface 404. In addition, the temporal directions ofpast 902 and future 904 can be swapped on either side of the focus 900.

Interaction Interface Descriptions

Referring to FIGS. 3 and 13, several interactive controls 306 supportnavigation and analysis of information within the visualizationrepresentation 12, as monitored by the visualization manger 300 inconnection with user events 109. Examples of the controls 306 are suchas but not limited to a time slider 910, an instant of focus selector912, a past time range selector 914, and a future time selector 916. Itis recognized that these controls 306 can be represented on the VI 202(see FIG. 1) as visual based controls, text controls, and/or acombination thereof.

Time and Range Slider 901

The timeline slider 910 is a linear time scale that is visibleunderneath the visualization representation 18 (including the temporal402 and spatial 400 domains). The control 910 contains subcontrols/selectors that allow control of three independent temporalparameters: the Instant of Focus, the Past Range of Time and the FutureRange of Time.

Instant of Focus

The instant of focus selector 912 is the primary temporal control. It isadjusted by dragging it left or right with the mouse pointer across thetime slider 910 to the desired position. As it is dragged, the Past andFuture ranges move with it. The instant of focus 900 (see FIG. 12) (alsoknown as the browse time) is the moment in time represented at thereference surface 404 in the spatial-temporal visualizationrepresentation 18. As the instant of focus selector 912 is moved by theuser forward or back in time along the slider 910, the visualizationrepresentation 18 displayed on the interface 202 (see FIG. 1) updatesthe various associated visual elements of the temporal 402 and spatial400 domains to reflect the new time settings. For example, placement ofEvent visual elements 410 animate along the timelines 422 and Entityvisual elements 410 move along the reference surface 404 interpolatingbetween known locations visual elements 410 (see FIGS. 6 and 7).Examples of movement are given with reference to FIGS. 14, 15, and 16below.

Past Time Range

The Past Time Range selector 914 sets the range of time before themoment of interest 900 (see FIG. 11) for which events will be shown. ThePast Time range is adjusted by dragging the selector 914 left and rightwith the mouse pointer. The range between the moment of interest 900 andthe Past time limit can be highlighted in red (or other colour codings)on the time slider 910. As the Past Time Range is adjusted, viewingparameters of the spatial-temporal visualization representation 18update to reflect the change in the time settings.

Future Time Range

The Future Time Range selector 914 sets the range of time after themoment of interest 900 for which events will be shown. The Future Timerange is adjusted by dragging the selector 916 left and right with themouse pointer. The range between the moment of interest 900 and theFuture time limit is highlighted in blue (or other colour codings) onthe time slider 910. As the Future Time Range is adjusted, viewingparameters of the spatial-temporal visualization representation 18update to reflect the change in the time settings.

The time range visible in the time scale of the time slider 910 can beexpanded or contracted to show a time span from centuries to seconds.Clicking and dragging on the time slider 910 anywhere except the threeselectors 912, 914, 916 will allow the entire time scale to slide totranslate in time to a point further in the future or past. Othercontrols 918 associated with the time slider 910 can be such as a “Fit”button 918 for automatically adjusting the time scale to fit the rangeof time covered by the currently active data set displayed in thevisualization representation 18. A scale control 918 includes a Fitcontrol 919, a scale-expand-contract controls 920, a step control 923,and a play control 922, which allow the user to expand or contract thetime scale. A step control 918 increments the instant of focus 900forward or back. The “playback” button 920 causes the instant of focus900 to animate forward by a user-adjustable rate. This “playback” causesthe visualization representation 18 as displayed to animate in sync withthe time slider 910.

Association Analysis Tools

Referring to FIGS. 1 and 3, association analysis functions 307 have beendeveloped that take advantage of the association-based connectionsbetween Events, Entities and Locations. These functions 307 are used tofind groups of connected objects 14 during analysis. The associations 16connect these basic objects 20, 22, 24 into complex groups 27 (see FIGS.6 and 7) representing actual occurrences. The functions 307 are used tofollow the associations 16 from object 14 to object 14 to revealconnections between objects 14 that are not immediately apparent.Association analysis functions 307 are especially useful in analysis oflarge data sets where an efficient method to find and/or filterconnected groups is desirable. For example, an Entity 24 maybe beinvolved in events 20 in a dozen places/locations 22, and each of thoseevents 20 may involve other Entities 24. The association analysisfunction 307 can be used to display only those locations 22 on thevisualization representation 18 that the entity 24 has visited orentities 24 that have been contacted.

The analysis functions 307 provide the user with different types of linkanalysis, such as but limited to:

-   -   1. Expanding Search        -   The expanding search function 307 allows the user to start            with a selected object(s) 14 and then incrementally show            objects 14 that are associated with it by increasing degrees            of separation. The user selects an object 14 or group of            objects 14 of focus and clicks on the Expanding search            button 920—this causes everything in the visualization            representation 18 to disappear except the selected items.            The user then increments the search depth and objects 14            connected by the specified depth are made visible the            display. In this way, sets of connected objects 14 are            revealed as displayed using the visual elements 410 and 412.    -   2. Connection Search        -   The Connection Search function 307 allows the user to            connect any two objects 14 by their web of associations 26.            The user selects any two objects 14 and clicks on a            Connection Search tool (not shown). The connection search            function 307 works by automatically scanning the extents of            the web of associations 26 starting from one of the objects            14. The search will continue until the second object 14 is            found as one of the connected objects 14 or until there are            no more connected objects 14. If a path of associated            objects 14 between the target objects 14 exists, all of the            objects 14 along that path are displayed and the depth is            automatically displayed showing the minimum number of links            between the objects 14.

It is recognized that the functions 307 can be used to implementfiltering via such as but not limited to criteria matching, algorithmicmethods and/or manual selection of objects 14 and associations 16 usingthe analytical properties of the tool 12. This filtering can be used tohighlight/hide/show (exclusively) selected objects 14 and associations16 as represented on the visual representation 18. The functions 307 areused to create a group (subset) of the objects 14 and associations 16 asdesired by the user through the specified criteria matching, algorithmicmethods and/or manual selection. Further, it is recognized that theselected group of objects 14 and associations 16 could be assigned aspecific name which is stored in the table 122.

Operation of Visual Tool to Generate Visualization Representation

Referring to FIG. 14, example operation 1400 shows communications 1402and movement events 1404 (connection visual elements 412—see FIGS. 6 and7) between Entities “X” and “Y” over time on the visualizationrepresentation 18. This FIG. 14 shows a static view of Entity X makingthree phone call communications 1402 to Entity Y from 3 differentlocations 410 a at three different times. Further, the movement events1404 are shown on the visualization representation 18 indicating thatthe entity X was at three different locations 410 a (location A,B,C),which each have associated timelines 422. The timelines 422 indicate bythe relative distance (between the elements 410 b and 410 a) of theevents (E1,E2,E3) from the instant of focus 900 of the reference surface404 that these communications 1404 occurred at different times in thetime dimension 432 of the temporal domain 402. Arrows on thecommunications 1402 indicate the direction of the communications 1402,i.e. from entity X to entity Y. Entity Y is shown as remaining at onelocation 410 a (D) and receiving the communications 1402 at thedifferent times on the same timeline 422.

Referring to FIG. 15, example operation 1500 for shows Events 140 boccurring within a process diagram space domain 400 over the timedimension 432 on the reference surface 404. The spatial domain 400represents nodes 1502 of a process. This FIG. 14 shows how a flowchartor other graphic process can be used as a spatial context for analysis.In this case, the object (entity) X has been tracked through theproduction process to the final stage, such that the movements 1504represent spatial connection elements 412 (see FIGS. 6 and 7).

Referring to FIGS. 3 and 19, operation 800 of the tool 12 begins by themanager 300 assembling 802 the group of objects 14 from the tables 122via the data manager 114. The selected objects 14 are combined 804 viathe associations 16, including assigning the connection visual element412 (see FIGS. 6 and 7) for the visual representation 18 betweenselected paired visual elements 410 corresponding to the selectedcorrespondingly paired data elements 14 of the group. The connectionvisual element 412 represents a distributed association 16 in at leastone of the domains 400, 402 between the two or more paired visualelements 410. For example, the connection element 412 can representmovement of the entity object 24 between locations 22 of interest on thereference surface 404, communications (money transfer, telephone call,email, etc.) between entities 24 different locations 22 on the referencesurface 404 or between entities 24 at the same location 22, orrelationships (e.g. personal, organizational) between entities 24 at thesame or different locations 22.

Next, the manager 300 uses the visualization components 308 (e.g.sprites) to generate 806 the spatial domain 400 of the visualrepresentation 18 to couple the visual elements 410 and 412 in thespatial reference frame at various respective locations 22 of interestof the reference surface 404. The manager 300 then uses the appropriatevisualization components 308 to generate 808 the temporal domain 402 inthe visual representation 18 to include various timelines 422 associatedwith each of the locations 22 of interest, such that the timelines 422all follow the common temporal reference frame. The manager 112 thentakes the input of all visual elements 410, 412 from the components 308and renders them 810 to the display of the user interface 202. Themanager 112 is also responsible for receiving 812 feedback from the uservia user events 109 as described above and then coordinating 814 withthe manager 300 and components 308 to change existing and/or create (viasteps 806, 808) new visual elements 410, 412 to correspond to the userevents 109. The modified/new visual elements 410, 412 are then renderedto the display at step 810.

Referring to FIG. 16, an example operation 1600 shows animating entity Xmovement between events (Event 1 and Event 2) during time slider 901interactions via the selector 912. First, the Entity X is observed atLocation A at time t. As the slider selector 912 is moved to the right,at time t+1 the Entity X is shown moving between known locations (Event1and Event2). It should be noted that the focus 900 of the referencesurface 404 changes such that the events 1 and 2 move along theirrespective timelines 422, such that Event 1 moves from the future intothe past of the temporal domain 402 (from above to below the referencesurface 404). The length of the timeline 422 for Event 2 (between theEvent 2 and the location B on the reference surface 404 decreasesaccordingly. As the slider selector 912 is moved further to the right,at time t+2, Entity X is rendered at Event2 (Location B). It should benoted that the Event 1 has moved along its respective timeline 422further into the past of the temporal domain 402, and event 2 has movedaccordingly from the future into the past of the temporal domain 402(from above to below the reference surface 404), since therepresentation of the events 1 and 2 are linked in the temporal domain402. Likewise, the entity X is linked spatially in the spatial domain400 between event 1 at location A and event 2 at location B. It is alsonoted that the Time Slider selector 912 could be dragged along the timeslider 910 by the user to replay the sequence of events from time t tot+2, or from t+2 to t, as desired.

Referring to FIG. 17, the visual reresentation 18 shows connectionvisual elements 412 between visual elements 410 situated on selectedvarious timelines 422. The timelines 422 are coupled to variouslocations 22 of interest on the geographical reference frame 404. Inthis case, the elements 412 represent geographical movement betweenvarious locations 22 by entity 24, such that all travel happened at sometime in the future with respect to the instant of focus represented bythe reference plane 404.

Referring to FIG. 18, the spatial domain 400 is shown as a geographicalrelief map. The timechart 430 is superimposed over the spatial domain ofthe visual representation 18, and shows a time period spanning fromDecember 3^(rd) to January 1^(st) for various events 20 and entities 24situated along various timelines 422 coupled to selected locations 22 ofinterest. It is noted that in this case the user can use the presentedvisual representation to coordinate the assignment of various connectionelements 412 to the visual elements 410 (see FIG. 6) of the objects 20,22, 24 via the user interface 202 (see FIG. 1), based on analysis of thedisplayed visual representation 18 content. A time selection 950 isJanuary 30, such that events 20 and entities 24 within the selection boxcan be further analysed. It is recognised that the time selection 950could be used to represent the instant of focus 900 (see FIG. 9).

It will be appreciated that variations of some elements are possible toadapt the invention for specific conditions or functions. The conceptsof the present invention can be further extended to a variety of otherapplications that are clearly within the scope of this invention. Havingthus described the present invention with respect to preferredembodiments as implemented, it will be apparent to those skilled in theart that many modifications and enhancements are possible to the presentinvention without departing from the basic concepts as described in thepreferred embodiment of the present invention. Therefore, what isintended to be protected by way of letters patent should be limited onlyby the scope of the following claims.

1. A method for creating a multidimensional visual representation of agroup of data elements having integrated temporal and spatialproperties, the data elements being included in the visualrepresentation as corresponding visual elements, the data elements ofthe group linked by at least one association, the method comprising thesteps of: assembling the group of data elements using the at least oneassociation; generating a spatial domain of the visual representation toinclude a reference surface for providing a spatial reference framehaving at least two spatial dimensions, the reference surface forrelating a first visual element representing a first data element of thegroup to a first location of interest in the spatial reference frame andrelating a second visual element representing a second data element ofthe group to a second location of interest in the spatial referenceframe; generating a temporal domain of the visual representationoperatively coupled to the spatial domain, the temporal domain forproviding a common temporal reference frame for the locations ofinterest, the temporal domain including a first time track coupled tothe first location of interest and a second time track coupled to thesecond location of interest, the first visual element positioned on thefirst time track and the second visual element positioned on the secondtime track, each of the time tracks configured for visually representinga respective temporal sequence of a plurality of the data elements ateach of the locations of interest of the reference surface; andassigning a connection visual element in the visual representationbetween the first visual element and the second visual element, theconnection visual element for representing a distributed association inat least one of the domains between the first visual element and thesecond visual element; wherein the visual representation is displayed ona user interface for subsequent interaction with user events.
 2. Themethod of claim 1, wherein the visual representation of the group ofdata elements is selected from the group comprising; a concurrent timeand geographic context, and a concurrent time and diagrammatic context.3. The method of claim 2 further comprising the step of configuring thereference surface for providing an instant of focus in the temporalreference frame for at least some of the temporal and spatial propertiesof the group of data elements, the instant of focus coupled to thelocations of interest of the reference surface, the instant of focusselected by the user events.
 4. The method of claim 3 further comprisingthe step of configuring the temporal reference frame for providing atime range selected from the group comprising; a past range of thetemporal sequence preceding the instant of focus, and a future range ofthe temporal sequence after the instant of focus.
 5. The method of claim4 further comprising the step of intersecting the first time trackthrough the first location of interest such that the past range of thefirst time track extends from one side of the reference surface and thefuture range of the first time track extends from the other side of thereference surface, the instant of focus located on the first time trackat the intersection point between the first time track and the referencesurface.
 6. The method of claim 5 further comprising the step ofarranging a first plurality of the visual elements along the first timetrack according to the times at which the visual elements occurred inthe temporal reference frame.
 7. The method of claim 6 furthercomprising the step of intersecting the second time track through thesecond location of interest such that the past range of the second timetrack extends from one side of the reference surface and the futurerange of the first time track extends from the other side of thereference surface, the second instant of focus located on the secondtime track at the intersection point between the second time track andthe reference surface.
 8. The method of claim 7 further comprising thestep of arranging a second plurality of the visual elements along thesecond time track according to the times at which the visual elementsoccurred in the temporal reference frame.
 9. The method of claim 6,wherein the location in the temporal reference frame of each of theplurality of visual elements on the time tracks is proportional to thedistance from the instant of focus associated with the referencesurface.
 10. The method of claim 9, wherein the time tracks arerepresented as timelines in the visual representation.
 11. The method ofclaim 10, wherein the temporal reference frame has a scale selected fromthe group comprising linear and logarithmic.
 12. The method of claim 2further comprising the step of maintaining an orientation of the timetracks with respect to the reference surface such that changes in theattitude of the reference surface in response to the user events resultsin a corresponding change in the orientation of the time tracks.
 13. Themethod of claim 12, wherein the orientation angle between the timetracks is 90 degrees.
 14. The method of claim 2 further comprising thestep of maintaining an orientation of the time tracks with respect tothe reference surface such that changes in the attitude of the referencesurface in response to the user events does not result in acorresponding change in the orientation of the time tracks.
 15. Themethod of claim 14, wherein the orientation of the time tracks is suchthat the length of the time tracks is maximized as perceived by a userof the user interface.
 16. The method of claim 2 further comprising thestep of overlapping a time chart on the first time track and the secondtime track, the time chart having a time axis and a spatial axis forrepresenting the temporal reference frame and at least one of thespatial dimensions respectively.
 17. The method of claim 16 furthercomprising the step of maintaining an orientation of the time chart withrespect to the reference surface such that changes in the attitude ofthe reference surface in response to the user events does not result ina corresponding change in the orientation of the time chart.
 18. Themethod of claim 17, wherein the time chart is represented as arectangular region.
 19. The method of claim 16 further comprising thestep of configuring the time chart for providing an instant of focus inthe temporal reference frame for at least some of the temporal andspatial properties of the group of data elements, the instant of focuscoupled to the locations of interest of the reference surface, theinstant of focus selected by the user events.
 20. The method of claim 19further comprising the step of configuring the time chart for providinga time range selected from the group comprising; a past range of thetemporal sequence preceeding the instant of focus, and a future range ofthe temporal sequence after the instant of focus.
 21. The method ofclaim 20, wherein the time chart has a superimposed grid forrepresenting the time axis and the spatial axis.
 22. The method of claim2, wherein types of the data elements are selected from the groupcomprising; entity, location, and event.
 23. The method of claim 22,wherein the event data element type represents an action taking place ata particular one of the locations of interest in the spatial referenceframe and at a particular time in the temporal reference frame.
 24. Themethod of claim 23, wherein the event data element type has dataproperties and display properties selected from the group comprising; ashort text label, description, location, start-time, end-time, generalevent type, icon reference, visual layer settings, priority, status,user comment, certainty value, source of information, anddefault+user-set color.
 25. The method of claim 22, wherein the entitydata element type represents an actor involved in a selected event. 26.The method of claim 25, wherein the entity data element type has dataproperties and display properties selected from the group comprising;short text label, description, general entity type, icon reference,visual layer settings, priority, status, user comment, certainty value,source of information, and default+user-set color.
 27. The method ofclaim 22, wherein the location data element type represents a locationwithin the spatial reference frame.
 28. The method of claim 27, whereinthe location data element type has data properties and displayproperties selected from the group comprising; position coordinates, alabel, description, color information, precision information, locationtype, non-geospatial flag and user comments.
 29. The method of claim 28,wherein the location data element type is selected from the groupcomprising; a physical location on a geospatial map, a physical locationas a node in a diagram, and a virtual location related to a geospatialmap.
 30. The method of claim 22, wherein the at least one associationdescribes a pairing between two or more of the data elements.
 31. Themethod of claim 30, wherein the connection visual element is a solidline representing a direct connection between the first visual elementand the second visual element.
 32. The method of claim 31, wherein thesolid line has a pointer for indicating a vector property of the visualconnection element.
 33. The method of claim 31, wherein the connectionvisual element is a trail represented as a series of connected linestracing known locations in the spatial reference frame of a selectedentity visual element over time represented by the temporal referenceframe.
 34. The method of claim 2 further comprising the step of updatingthe visual elements in the visual representation in response to the userevents.
 35. The method of claim 34, wherein the user events aregenerated in response to manipulation by the user of an interactivecontrol for modifying visual properties of the visual representation.36. The method of claim 35, wherein the interactive control is selectedfrom the group comprising a time range selector and an instant of focusselector.
 37. The method of claim 35 further comprising the step ofanimating the display of at least one of the visual elements of thevisual representation in response to the manipulation of the interactivecontrol.
 38. The method of claim 37, wherein the display of theconnection visual element of the visual representation is animated inresponse to the manipulation of the interactive controls.
 39. The methodof claim 38, wherein the connection visual element is coupled to themovement of an entity visual element across the visual representationbetween the first location of interest and the second location ofinterest, entity visual element representing an actor involved in aselected event.
 40. The method of claim 35 further comprising the stepof applying a filtering function to the visual elements and the at leastone related association to select a subgroup thereof.
 41. The method ofclaim 40 further comprising the step of selecting the subgroup accordingto a method selected from the group comprising; criteria matching,algorithmic methods, and manual selection.
 42. The method of claim 40further comprising the step of processing the selected subgroup ofvisual elements to change the presentation of the visual representationselected from the group comprising; highlighting the subgroup andremoving the subgroup.
 43. A system for creating a multidimensionalvisual representation of a group of data elements having integratedtemporal and spatial properties, the data elements being included in thevisual representation as corresponding visual elements, the dataelements of the group linked by at least one association, the systemcomprising: a visualization manager for assembling the group of dataelements using the at least one association and for assigning aconnection visual element in the visual representation between a firstvisual element representing a first data element of the group and asecond visual element representing a second data element of the group; aspatial visualization component configured for generating a spatialdomain of the visual representation to include a reference surface forproviding a spatial reference frame having at least two spatialdimensions, the reference surface for relating the first visual elementto a first location of interest in the spatial reference frame andrelating the second visual element to a second location of interest inthe spatial reference frame; and a temporal visualization componentconfigured for generating a temporal domain of the visual representationoperatively coupled to the spatial domain, the temporal domain forproviding a common temporal reference frame for the locations ofinterest, the temporal domain including a first time track coupled tothe first location of interest and a second time track coupled to thesecond location of interest, the first visual element positioned on thefirst time track and the second visual element positioned on the secondtime track, each of the time tracks configured for visually representinga respective temporal sequence of a plurality of the data elements ateach of the locations of interest of the reference surface; and whereinthe connection visual element represents a distributed association in atleast one of the domains between the first visual element and the secondvisual element such that the visual representation is displayed on auser interface for subsequent interaction with user events.
 44. Acomputer program product for creating a multidimensional visualrepresentation of a group of data elements having integrated temporaland spatial properties, the data elements being included in the visualrepresentation as corresponding visual elements, the data elements ofthe group linked by at least one association, the computer programproduct comprising: a computer readable medium; a visualization modulestored on the computer readable medium for assembling the group of dataelements using the at least one association and for assigning aconnection visual element in the visual representation between a firstvisual element representing a first data element of the group and asecond visual element representing a second data element of the group; aspatial visualization module stored on the computer readable medium forgenerating a spatial domain of the visual representation to include areference surface for providing a spatial reference frame having atleast two spatial dimensions, the reference surface for relating thefirst visual element to a first location of interest in the spatialreference frame and relating the second visual element to a secondlocation of interest in the spatial reference frame; and a temporalvisualization module stored on the computer readable medium forgenerating a temporal domain of the visual representation operativelycoupled to the spatial domain, the temporal domain for providing acommon temporal reference frame for the locations of interest, thetemporal domain including a first time track coupled to the firstlocation of interest and a second time track coupled to the secondlocation of interest, the first visual element positioned on the firsttime track and the second visual element positioned on the second timetrack, each of the time tracks configured for visually representing arespective temporal sequence of a plurality of the data elements at eachof the locations of interest of the reference surface; and wherein theconnection visual element represents a distributed association in atleast one of the domains between the first visual element and the secondvisual element such that the visual representation is displayed on auser interface for subsequent interaction with user events.